Estimating the range to a mobile object using a monocular camera fixed to a moving platform has, in general, remained an unsolved problem for the past three decades. An effective solution to this problem can be used in autonomous collision detection and avoidance applications for unmanned vehicles, especially unmanned aerial vehicles and surface vehicles. The technology can also be used to provide situational awareness information to manned air and water crafts moving in dynamic environments. In this disclosure, we present a novel approach to compute the range or distance to a dynamic object using a sequence of monocular images captured from a moving sensor platform. In what follows, we shall use the term “intruder” to describe the dynamic object whose range is to be estimated.
Depending on the sensor resolution, imaging sensors can provide very accurate estimates of the relative bearing angle to the intruder. At large distances, an intruder may occupy only a few pixels on an image. The classical approach is to treat the intruder as a point target and use bearing angle measurements to the tracked point. However, this approach is, in general, not sufficient due to the inherent unobservability of the intruder dynamics. We use an example to illustrate this problem.
Let us assume that the sensor platform is initially located at (0,0) in a two-dimensional plane and moves along the positive y-axis with velocity given by vs. Let us assume that the intruder initially located at (x0, y0) is moving with velocity (u, v) where u and v represent the X and Y components of the velocity. The position of the sensor at any time t is given by (0, vst). The position of the intruder at time t is given by (x0+ut, y0+vt). The bearing angle to the intruder as measured from the sensor at any given time is, therefore, given by arctan (y0+(v−vs)t)/(x0+ut). Now, let us assume there is another intruder located initially at (kx0, ky0) moving with velocity (ku, kv−(k−1)vs), where k is a positive constant and not equal to 1. Now, notice that the relative bearing to the second intruder measured from the sensor at any time t is given by arctan (ky0+(kv−(k−1)vs)t−vst)/(kx0+kut) which is equal to arctan (ky0+kvt−kvst)/(kx0+kut)=arctan (y0+(v−vs)t)/(x0+ut). Therefore, two intruders with distinct trajectories (k≠1) generate the same bearing angle measurement at the sensor. This clearly illustrates the inherent observability problem present in bearing-only tracking. Please refer to FIG. 1 for a specific example. In this case, (x0, y0)=(500,500), k=2, vs=100, u=−50 and v=50. The sensor position is shown by red circles. The first intruder's position is represented by blue plus signs. The second intruder (position denoted by black asterisks) is moving with velocity equal to (−50k, 50k−(k−1)100)=(−100,0). The angles θ(0), θ(1) and θ(2) denote the relative bearing to the two intruders at time instants 0, 1 and 2. Notice that both intruders subtend the same relative bearing angles with respect to the sensor, but k≠1 signifying that the two intruder trajectories are distinct as can be seen in the figure.
The conditions of unobservability in a bearing-only tracking problem have been extensively studied since the late 1970's. This body of research has established that the intruder state is observable, in general, only if the order of sensor dynamics is greater than the intruder dynamics. For an intruder moving with constant velocity, this implies that the sensor dynamics must involve an acceleration component. In the event that the sensor dynamics are not of a higher order, the sensor platform must execute a deliberate maneuver involving a higher order dynamics component to be able to estimate the range. With the growing use of Unmanned Aerial Vehicles (UAVs) in recent years, such a “maneuver-based” approach has been proposed as a solution to the passive camera based Sense and Avoid (SAA) problem: upon detecting an intruder, the UAV maneuvers in order to triangulate and resolve the position of the intruder. However, a maneuver-based approach is undesirable in many ways especially in military operations. It may lead to waste of fuel, loss in mission performance, and is in general bad airmanship.
Other research deals with choosing the right coordinate frames and filters for bearings-only tracking from the point of view of stability and unbiasedness. For example, an Extended Kalman Filter (EKF) applied to bearings-only target tracking has been theoretically analyzed and has established the reasons for the filter bias and instability. A modified polar coordinate system based Kalman Filter has also been proposed to separate the observable and unobservable dynamics. Further, a bank of EKFs each with a different initial range estimate referred to as the range-parameterized (RP) tracker has been proposed and shown to perform better than classical EKF implementations. More recently, particle filter solutions to bearings-only tracking problem are receiving considerable attention and they have been implemented to track both maneuvering and non-maneuvering targets. Particle filter and RP EKF have been compared and it has been shown that the particle filter is only marginally better than the Range Parameterized EKF but is considerably more robust to errors in the initial target range.
This disclosure presents a novel passive image based ranging approach that does not require the sensor platform to maneuver. Analytical results prove that the approach can be used to estimate the range to an intruder without any sensor maneuver under very general conditions. Tests conducted on real flight-test data have demonstrated the practicality of the approach.
The invention in this disclosure concerns a novel method to estimate the range to a moving rigid body from a mobile platform using monocular passive cameras such as electro-optical or infra-red camera mounted on the platform. The primary application of the proposed technology is in the area of collision detection and avoidance for unmanned vehicles. Unmanned aerial, surface and ground vehicles equipped with cameras can use the proposed invention to estimate the range to other moving objects in their vicinity. The range information can then be used to determine the trajectory of the moving objects and maintain a safe distance from them.
Generally, in a computerized system including a camera mounted on a moving vehicle, wherein the camera acquires consecutively in real time a plurality of images of a moving object within a field of view of the camera, a method for determining a range of said moving object from said moving vehicle comprises the steps of:
(a) detecting the moving object (intruder) within each of said plurality of images;
(b) identifying two feature points p1 and p2 on said detected object where p1 and p2 satisfy a certain geometric relationship with the velocity vector of the detected object including but not limited to that the two feature points represent a leading point of the detected object and a trailing point of the detected object; and
(c) recursively calculating the range to the object based on changes in the positions of the feature points p1 and p2 in the sequential images and further based on the assumption that the geometric relationship mentioned under item (b), including but not limited to the assumption that the detected object is traveling in a direction along a line connecting feature points p1 and p2, is valid
The key aspect of the invention lies in the premise that if you can assume pure translational movement and assume a direction of motion based on the image, then you can recursively calculate the range without the order of sensor dynamics being greater than the intruder dynamics, i.e. without any extraneous maneuvering.
Accordingly, among the objects of the instant invention are: the provision of a maneuverless passive ranging method.
Another object of the invention is the provision of a ranging method which extracts feature points on a detected object and provides a geometric relationship between the spatial positions of the feature points and the direction of velocity of the object.
Other objects, features and advantages of the invention shall become apparent as the description thereof proceeds when considered in connection with the accompanying illustrative drawings.